cache benchmark runs#417
Merged
Merged
Conversation
vastzuby
force-pushed
the
AUTO-1452-benchmarks-run-cache
branch
from
June 24, 2026 01:09
e97b98e to
9ebf5d5
Compare
vastzuby
changed the title
vastzuby
force-pushed
the
AUTO-1452-benchmarks-run-cache
branch
from
June 30, 2026 19:54
88582c7 to
abeda07
Compare
| Template is matched client-side: rows carry template_hash or template_id | ||
| depending on how the benchmarked workergroup was created. | ||
| """ | ||
| query = { |
Contributor
There was a problem hiding this comment.
We should include a limit on this query, since it could potentially return 1000s of rows for common template/num_gpu/gpu_name combos (i.e. ComfyUI on 1x5090)
vastzuby
changed the title
vastzuby
changed the title
LucasArmandVast
approved these changes
Jul 15, 2026
Reuse the benchmarks table as a cross-user cache for `vastai run benchmarks`: before renting, look for a recent (<=30d) row matching the same template + GPU + count and serve its median measured perf instead of renting and re-measuring. - Perf-only cache: cached rows carry no $/hr, since a cached row's price would come from a different machine than the one benchmarked; $/hr and perf/$ render "-" for cached rows. - Show the perf spread (median, range, n, age) so real machine-to-machine variance is visible rather than a single misleading number. - On a cache hit, prompt to reuse the result or run a fresh benchmark; -y/--raw are non-interactive and reuse the cache. --no-cache always re-measures. - Flag cache hits that have no current matching offers as not rentable. - Fix endpoint_name: drop the parens around the uuid suffix; the backend rejects shell metacharacters in endpoint_name, so any real rental 400'd (pre-existing bug in the base command). - Lower the default --timeout from 60m to 30m. - search_benchmarks type hint accepts dict queries. - Drop leading underscores from module helpers/constants to match the rest of the cli commands; tidy comments, docstrings, and constant grouping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JKE6tJgMRhV5tcLtYBYKnQ
lookup_cached_benchmark pulled every perf row for a gpu_name/num_gpus combo and matched the template client-side, discarding the rest: 3275 rows for a common combo like 1x RTX 5090, once per GPU in the sweep. The benchmarks table is indexed on (template_hash, last_update desc) and (template_id, last_update desc), so neither index could be used. Filter on the template columns instead, newest-first and capped, which is index-backed and returns ~2 rows. A workergroup is created with either template_id or template_hash, so a row can carry either; query once per column and dedupe. search_benchmarks grows order/limit params to pass these through.
vastzuby
force-pushed
the
AUTO-1452-benchmarks-run-cache
branch
from
July 16, 2026 02:42
d225a4a to
520629e
Compare
test_prune_failure_does_not_fail_the_update still asserted env/bin/vastai after the XDG refactor (#452) renamed the managed-install layout to current/. The two sibling success tests were updated; this one was missed, so master went red once #447 and #452 both landed. perform_update builds current/, so match it.
#455 renamed the aggregator job's reported name from unit-and-integration to unit-and-integration-check while remediating CVEs. Branch protection requires a context literally named unit-and-integration, which the matrix jobs never emit (they report 'unit-and-integration (ubuntu-latest)' etc), so with the aggregator renamed nothing satisfied the rule and every PR hung on an expected check. Restore the bare name the rule expects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
overview
vastai run benchmarksrents a real GPU, measures perf, and writes a record to the benchmarks table.this PR will use that table as a cache so that we dont have run duplicate benchmarks.
a benchmark is duplicate if it has the same gpu type, num_gpus, template, and ran in past 30 days
caching
cache hit -> skip the rental and return the median of the reported perfs.
cache miss -> run a new benchmark
note that the cache reports the perf while a real benchmark will return the perf/$ but we dont know the gpu price yet for historical benchmarks.
edge cases
if no offers available, we flag to user ("no offers available to rent right now")
future work
update the benchmarks table to track the price of a benchmarked gpu.
that way, we can return the perf/$ for cached responses.
also includes unrelated test fix where we refactored env -> current in perform update() but test file was still asserting the old layout (env/bin/vastai)
unrelated test fix
The XDG-layout refactor (#452) renamed env/ → current/ and but test_prune_failure_does_not_fail_the_update (added in #447) still asserted the old env/ path